Tuning and evolution of support vector kernels

Kernel-based methods like Support Vector Machines (SVM) have been established as powerful techniques in machine learning. The idea of SVM is to perform a mapping φ from the input space to a higher-dimensional feature space using a kernel function k, so that a linear learning algorithm can be employed. However, the burden of choosing the appropriate kernel function is usually left to the user. It can easily be shown, that the accuracy of the learned model highly depends on the chosen kernel function and its parameters, especially for complex tasks. In order to obtain a good classification or regression model, an appropriate kernel function must be used. Design and hand-tuning of kernel functions can be timeconsuming and requires expert knowledge. To circumvent these obstacles for the ’non-expert’ data mining user, which may hinder the wider use of SVM in data mining, we present two solutions for optimizing kernel functions: (a) automated hyperparameter tuning of kernel functions and (b) evolving new kernel functions by Genetic Programming (GP). We review state-of-the-art techniques for both approaches, comparing their different strengths and weaknesses. Special attention is drawn to Sequential Parameter Optimization (SPO) for tuning, as this method also allows a statistical evaluation and understanding of the respective influences of the parameters. A tuned kernel can improve the trained model, if standard kernels are insufficient for achieving a good transformation. We apply tuning to SVM kernels for both regression and P. Koch · O. Flasch · T. Bartz-Beielstein ·W. Konen Cologne University of Applied Sciences, Institute of Computer Science, Germany E-mail: {patrick.koch, oliver.flasch, thomas.bartz-beielstein, wolfgang.konen}@fh-koeln.de Bernd Bischl University of Dortmund, Faculty of Statistics, Germany E-mail: [email protected] classification. In fact often a good kernel function is missing for real-world problems. We compare standard kernels with hand-tuned parameters to SPO-tuned standard kernels and to GP-generated custom kernels for these problems.